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Foreword 



Over the last 10 years the Center for Substance Abuse Treatment (CSAT) has 
accumulated a great deal of experience in substance abuse treatment evaluation implemented 
through coordinating centers, cross-site efforts, and national studies. The importance and value 
of integrating ongoing evaluation activity into a system for treating substance abuse problems is 
widely recognized by treatment providers and by CSAT. Also widely recognized, however, is 
that current evaluation generated knowledge and practice are often under-utilized, due in part to 
the lack of an integrated approach to capturing information with which to measure treatment 
outcomes and improve the treatment process. CSAT recognizes that such an integrated 
evaluation approach will more effectively support its knowledge generating activities. 

Based on a decade of evaluation experience, CSAT has developed the Integrated 
Evaluation Methods (IEM) Package, a series of conceptual and methodological applications, 
including concept papers, technical assistance materials, and analytic tools, to enhance CSAT- 
funded evaluation activities. Products in the IEM Package are organized within an evaluation 
framework constructed on the basis of accumulated experiences among internationally known 
treatment service evaluation professionals. Thus, the framework is based upon evaluation 
strategies, structures and approaches appropriate for substance abuse treatment evaluators and 
providers. The framework follows a standard set of evaluation activities: planning, selecting a 
design, developing data requirements and collection instruments, collecting and analyzing the 
data, and reporting the evaluation findings. (A summary description of the IEM Package is 
contained in the Appendix to this document.) 

This document, along with its two companion documents, A Guide to Process Evaluation 
for Substance Abuse Treatment Services and A Guide to Selecting an Outcome Evaluation 
Design for Substance Abuse Treatment Evaluations, is specifically aimed at supporting the 
design stage of the evaluation process. This document provides a definition of logic models and 
discusses their use in treatment services planning and evaluation. Taken together, these three 
documents are intended to assist substance abuse treatment professionals to plan and conduct 
scientifically valid, and therefore meaningful, evaluation activities. 

Sharon Bishop 
Director 

National Evaluation Data and Technical Assistance Center 
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I. Introduction 



The Center for Substance Abuse Treatment (CSAT) supports the integration of ongoing 
evaluation within substance abuse treatment activities so as to demonstrate treatment service 
effectiveness and to improve treatment services and their outcomes. To this end, CSAT 
recommends the use of state-of-the-art evaluation methods and tools in planning, designing, and 
implementing treatment services evaluations. This paper discusses one of these tools: the logic 
model. The logic model provides the linkage among all of the evaluation activities and ensures 
the integration of process and outcome evaluation results. 

1. CONTEXT FOR THE LOGIC MODEL DOCUMENT 

CSAT’s major evaluation goals are to: (1) increase knowledge about substance abuse 
treatment services; (2) improve treatment services by applying knowledge gained through 
knowledge development and application (KD&A) activities; (3) develop analytic methods and 
approaches for use in knowledge-generating activities; and (4) develop substance abuse treatment 
analysis databases. To meet these goals, CSAT has been sponsoring KD&A initiatives including 
activities that focus on homelessness, marijuana use and treatment, managed care, women and 
violence, and opioid treatment, as well as the replicability of exemplary treatment approaches 
(e.g., methamphetamine treatment) and the evaluation of best practices for targeted populations 
(e.g., exemplary adolescent treatment). 

CSAT’s evaluation experiences have reinforced the fact that substance abuse treatment 
evaluation involves a standard set of tasks that generally occur in the following order: 

■ Planning the evaluation, which includes setting the evaluation goals and objectives 
that determine the overall parameters of the evaluation 

■ Selecting the evaluation design, which sets forth the overall strategy for establishing 
the evaluation questions, measurement approach, and generalizability of findings 

■ Developing the data requirements, which flow from the evaluation questions and 
measures and include SDU, clinician, cost, and client data 

■ Developing data collection instruments, which are based on the data requirements 
and are developed or selected from a standard inventory of instrumentation 

■ Collecting the data, which includes the development of data management processes 
and tools including quality control procedures, and collecting the data 
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Introduction 



■ Analyzing the data, which involves developing an analysis plan and conducting 
multiple levels of comparison; the analysis process is governed by the analysis plan 
and intended products and target audience(s) 

■ Reporting the evaluation findings, which includes evaluation knowledge 
dissemination and application within field. 

CSAT has directed the development of evaluation concepts, methods, and tools to support these 
evaluation tasks. The evaluation tasks and corresponding evaluation methods products are 
summarized in Exhibit I of the appendix to this document. As shown, the use of logic models in 

A ~ 

CSAT evaluations is part of the second stage in the evaluation process: selecting the evaluation 
design. A full discussion of the CSAT evaluation analytic framework and the other evaluation 
concepts and tools is presented in the concept paper: Integrated Evaluation Methods: A Guide 
for Substance Abuse Treatment Knowledge Generating Activities. This document, taken together 
with the other evaluation methods products in Exhibit I, is known as the Integrated Evaluation 
Methods Package. The documents that comprise the Package are being made available on the 
Caliber Associates NEDS contract Web site at http://neds.calib.com. 

2. HOW THIS PAPER IS ORGANIZED 

The paper is divided into six sections. Following the introduction, the paper provides a 
definition of logic models and discusses their use in treatment services planning and in treatment 
services evaluation. The paper concludes with a description and examples of using logic models 
to develop data maps that specify evaluation questions, measures, and variables. 



O CSAT\CTRT_END\1EM\C0NCEPT\L0GICMDL\DATA\L0GICMDL.WPD 



NEDTAC, Page 2 



II. Description of the Logic Model 



The following paragraphs provide a definition of logic models and describe the logic 
model components and uses. 

1. DEFINITION 

A logic model is a descriptive, graphic representation of substance abuse treatment 
services and how they are supposed to work. A logic model includes a succinct, logical chain of 
statements that link substance abuse problems, treatment service interventions, and expected 
treatment outcomes. Logic models incorporate the theoretical relationships among the source or 
cause of the substance abuse problem, the design of the treatment service intervention, and the 
expected treatment results (i.e., outcomes). Logic models are conceptually straight-forward but, 
for substance abuse treatment, can be extremely complex because there are numerous, sometimes 
competing, theories as to the causes of substance abuse, the most effective treatment 
interventions, and the multiple short-term and long-term outcomes (Kumfer et al., 1993). For 
any given treatment approach and evaluation strategy, these issues can be identified and included 
in the logic model for that specific effort. 

2. LOGIC MODEL COMPONENTS 

A logic model typically consists of four components: 

■ Conditions and context in which the substance abuse treatment services operate, 
including the target population characteristics, the community resources, and the 
regulations and policies that govern the treatment services operations 

■ Activities and services offered as part of the substance abuse treatment 

■ Short-term outcomes which are immediately expected to result from the treatment 
services 

■ Long-term outcomes (impacts) which are expected, and which should correspond to 
the treatment service goals. 

The linkages for the conditions, activities, short-term outcomes, and long-term outcomes are the 
theoretical underpinnings which guide the treatment services design, the implementation, and the 
evaluation (Conrad et al., 1998). 
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Description of the Logic Model 



Logic models are most effective when presented graphically because the graphic 
presentation clearly establishes the interrelationships of the four components. Logic models are 
typically a series of boxes in columns, one column each for the conditions, activities, short-term 
outcomes, and long-term outcomes. Horizontal lines reflect the interrelationships among the 
problem conditions, the treatment service activities, and the expected treatment outcomes. A 
simplified hypothetical logic model for a treatment service for pregnant women is illustrated in 
Exhibit II- 1. 



Exhibit II-l 

Example of Simplified Logic model 
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3. USES OF LOGIC MODELS 

There are numerous applications for logic model techniques; within the Federal 
evaluation environment, four applications predominate. These include: (1) Federal/funding 
agency grants management; (2) substance abuse treatment services design and management; (3) 
substance abuse treatment services evaluation; and (4) knowledge generation about substance 
abuse treatment effectiveness and identification of exemplary or best practices. 

One of the assumptions underlying the IEM is that the use of logic models should be a 
key component in reviewing grant and cooperative agreement applications, monitoring projects, 
and designing and implementing evaluations of knowledge-generating initiatives. SAMHSA, 
including CSAT, specifies in the Guidance for Applicants (GFAs) that grant and cooperative 
agreement applicants include a logic model within the grant application. The logic model assists 
the technical review of the application and is then used by project officers to assess project 
implementation and the extent to which the project remains faithful to the project design. Project 
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Description of the Logic Model 



logic models within a grant program also are used to determine the similarities and differences 
across projects as part of a program-wide evaluation (Conrad et al., 1 998). 

Logic models also assist substance abuse treatment services design and management and 
substance abuse treatment services evaluation. Traditionally, logic models are developed in 
advance of the treatment service design and the evaluation design. Integration Evaluation 
Methods (IEM) efforts expand on this traditional use by incorporating evaluation activities and 
products within the logic model process. When developed at the outset, the logic model ensures 
that the treatment services staff and the evaluation staff have a shared understanding of the 
purpose, components, and expected results of the treatment services, since the logic model: 

■ Clearly identifies treatment service goals, objectives, activities, and desired outcomes 

■ Clarifies assumptions and relationships between treatment services efforts and 
expected results 

■ Helps to specify what to measure through evaluation, when, and why 

■ Aids in determining how to link process evaluation measures and outcome evaluation 
measures 

■ Guides the assessment of underlying assumptions and facilitates self-correction of the 
treatment services (Kumpfer et al., 1993). 

In addition, developing the treatment services logic model jointly by treatment services and 
evaluation staff assists in building consensus and a common understanding of treatment service 
provision. The value of the logic model is maximized since the design and development of 
treatment services and treatment service evaluation are fully coordinated. 

The next two sections of this paper describe the value of logic models for treatment 
service planning and for treatment service evaluation planning. In fact, the development of logic 
models for treatment services and their evaluations must be coordinated. The interrelationship of 
the treatment services logic model and the evaluation logic model is diagrammed in Exhibit II-2. 
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EXHIBIT II-2 

INTERRELATIONSHIP OF LOGIC MODELS FOR PLANNING 
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III. Using Logic Models to Plan Knowledge-generating 

Treatment Services 



Logic models are vital to the design and development of substance abuse treatment 
services and useful for services management and monitoring, and for knowledge-generating 
activities specific to a knowledge development or knowledge application activity. Since 
substance abusing behaviors are influenced by a variety of factors, treatment services typically 
attempt to address multiple factors, simultaneously. By assessing the conditions (including the 
target population), framing the problem statement s), designing the treatment services, and 
specifying the expected outcomes, up front, the treatment services will be designed coherently. 
The logic model then supports the management of the treatment services by: (1) providing a tool 
to assess implementation so as to ensure that the services are implemented as planned; 

(2) maintaining the integrity of the treatment services to the design, overtime; and (3) monitoring 
treatment service costs. 

A framework for developing a treatment services logic model using treatment services for 
pregnant women as an example is presented in Exhibit III-l. The principles that guide the 
development of the treatment services logic model are described, below. 

1 . CONDITIONS AND CONTEXT FOR KNOWLEDGE-GENERATING 

TREATMENT SERVICES 

In designing substance abuse treatment services, a first step is to identify the conditions 
for the treatment services, define the “problem” to be addressed by the treatment interventions 
and the knowledge-generating activity, and specify the assumptions on which the treatment 
services design will be based. 

Conditions which may influence the type of treatment services that should be offered 
and/or that may affect the treatment services outcomes include demographic characteristics and 
the substances being abused. For example, gender has been found to influence treatment service 
access, entry, retention, and outcomes. It also influences the treatment services that are needed to 
maximize treatment success. Therefore, treatment services for women should include targeted 
outreach efforts and specific services that address women’s unique medical and familial needs. 
This also is true for other treatment populations such as adolescents, injection drug users, and 
others. Also, different substance addictions respond differently to different treatments. For 
example, some addictions respond better to pharmacological treatment while others respond 
better to individual and group therapies. To ensure that treatment services are designed 
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exhibit ill-l 

Sample Logic Model for Substance Abuse Treatment Services for Pregnant Women 
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Using Logic Models to Plan Knowledge-Generating Treatment Services 



appropriately, a literature review should inform the design based on tested theories or practices 
that have proved successful for a specific group or substance of abuse. 

The problem statement should include behaviors and conditions that can be affected by 
the knowledge-generating treatment intervention. For example, lack of financial resources, or 
poverty, may be a factor in an individual’s substance abuse but treatment services cannot treat 
poverty, per se. Rather, treatment services can provide education and job skills training which, 
in turn, can lead to improved financial conditions. 

Another important condition for substance abuse treatment services is the availability of 
resources to fund the services and community resources to support the services. Federal, state, 
and other resources are generally available but may be reserved for specific populations and/or 
types of substances being abused. Also, existing community substance abuse treatment resources 
may suggest strategies for inter-agency networking, coordination, and cooperation. In addition, 
funding sources typically have associated regulations and policies which should be accounted for 
by the substance abuse treatment services design. 

A needs assessment is frequently used to identify the type and extent of existing problems 
within the community, the services available, and the unmet needs. A needs assessment is a 
process to determine the need, which can be defined as the gap between the problem and existing 
resources to address the need (Linney et al., 1991). 

2. KNOWLEDGE-GENERATING TREATMENT SERVICES ACTIVITIES 

A substance abuse treatment services logic model requires the specification of two types 
of activities: inputs to the treatment activities and service implementation. Inputs include the 
specification of treatment service goals and objectives; identification of treatment models; 
establishment of linkages with other community resources; and the treatment resources including 
funding, staffing, facilities, and costs. Specifying the services implementation for the logic 
model includes a listing of the specific services to be provided. Again, these services should 
logically flow from the conditions (including target population needs), the project goals and 
objectives, and the treatment model being adopted (Devine et al., revised 1999). 
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3. KNOWLEDGE-GENERATING TREATMENT SERVICES OUTCOMES 

The logic model should be carefully crafted so as to appropriately anticipate and 
distinguish between short-term outcomes and long-term outcomes. The long term or ultimate 
outcomes of the treatment services are sometimes shown in the logic model as the treatment 
service goals. However they are stated, it is advantageous to identify stages of desired outcomes 
and to differentiate between short-term and long-term treatment services outcomes. 

For example, in residential treatment services designed to treat substance-abusing 
pregnant women, the short-term treatment outcome effects may be an increased community 
awareness, an increased number of mothers who receive prenatal care in the first trimester of the 
pregnancy, an increased number of healthy births, a reduced number of low birth weight babies, 
and a decreased number of women who use alcohol and drugs. The long-term outcomes or 
ultimate treatment service goals may be to reduce infant morbidity and mortality, reduce mental 
defects in newborns, maintain parental sobriety, and increase family reunification. In addition to 
short-term and long-term client outcomes, the knowledge-generating treatment service may be 
designed to have an impact on the community. For example, the location of the treatment service 
may be determined by the community’s greatest need for additional treatment services. An 
expected short-term outcome may be a reduction in the number of people on treatment service 
waitlists. The longer-term community or systemic outcomes may be reduced health care costs 
since community residents are receiving substance abuse treatment services more promptly, and 
the outcome of the treatment service is reduced need for primary health care. 

It is important to state outcomes with as much specificity as possible so that they can be 
measured. Outcomes should explain what problems the substance abuse treatment services are 
attempting to eliminate, and, where possible, should include time frames and conditions under 
which the outcomes are expected to occur. For example, a short-term outcome for the residential 
treatment services for women may be to maintain 120 consecutive days in the treatment services. 
A long-term outcome for these services may be to maintain 2 years of abstinence. 
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IV. Using Logic Models for Evaluating 
Knowledge-generating Activities 



Logic models are essential to interpreting evaluation findings. The evaluator needs a 
clear understanding of the treatment services goals, implementation sequences, and expected 
links among the treatment population characteristics, services, and expected benefits so as to 
accurately interpret the evaluation results (Wholey, 1 979). An example of a logic model 
framework for use in developing a substance abuse treatment services evaluation is presented in 
Exhibit IV- 1 and described below. 

1 . conditions and context for the knowledge-generating 

evaluation 

Knowledge-generating substance abuse treatment services evaluations must understand 
the environment in which the treatment services are operating and the new knowledge that can be 
obtained. The conditions component of the logic modeling process assists in planning the 
evaluation of the treatment population and the environment. First, the evaluation can assess the 
target population in terms of prevalence and treatment needs. Services for pregnant women, for 
example, would be under-utilized if offered in communities with few substance-abusing pregnant 
women. Further, an evaluation designed to use random assignment must assess prevalence so as 
to design the client flows for treatment and ensure that the client flows for the treatment 
alternatives will yield sufficient sample sizes. 

The evaluation also must be fully informed about the local treatment resources, funding 
sources, and associated regulations and policies. Funding sources and/or alternative community 
treatment resources may influence the implementation of the treatment services being evaluated 
and the evaluation must take these confounding variables into account when interpreting the 
evaluation results. 

Logic models also are useful in assessing the evaluability of substance abuse treatment 
services. Evaluability assessments are the pre-evaluation analyses that help to ensure that an 
evaluation will be technically feasible and capable of answering the evaluation and research 
questions important to decision makers. The evaluability assessment lays the groundwork for a 
successful evaluation by: 
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Exhibit IV-1 

Sample Logic Model for evaluation of Knowledge-Generating Treatment Services 
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■ Establishing agreement with the treatment service provider and the evaluator on the 
hypothesized causal links between the treatment services design and the intended 
intermediate and long term outcomes 

■ Providing an organizing framework for measurement and data collection so as to 
ensure that all data necessary to test the hypotheses and meet decision-maker’s needs 
will be collected 

■ Building consensus between the funder, the treatment director, the clinical director, 
and the evaluators on data collection procedures and ensuring that adequate data 
collection systems are in place or will be established. 

During the evaluability assessment, the evaluator examines the context for the substance abuse 
treatment services and the evaluation to see if a rigorous, objective evaluation is possible and the 
logic modeling process supports this activity (Wholey, 1994; Conrad et al., 1998). 

2. KNOWLEDGE-GENERATING SUBSTANCE ABUSE TREATMENT SERVICES 

The evaluation logic model supports the evaluation of the substance abuse treatment 
services activities in five ways: (1) clarifying treatment service goals; (2) identifying the 
underlying treatment theories; (3) providing a framework for organizing the process evaluation; 
(4) providing a framework for integrating the process and outcome analyses; and (5) ensuring the 
knowledge-generating goals and objectives are met. 

Clarifying treatment service goals. Most substance abuse treatment services have 
multiple treatment goals. Logic models assist the evaluation process by identifying the treatment 
services goals that are most important to the evaluation and for which there are sound 
measurement methods. For knowledge-generating activities, the new knowledge to be gained is 
the primary treatment service goal. 

Identifying underlying treatment theories. Many treatment services are developed 
based on the experiences of the treatment providers. Implicit in these treatment services, 
however, are underlying theories and hypotheses about the characteristics of the treatment 
population, the substances that are abused, and effective methods for addressing specific 
population and/or substance characteristics. An evaluation that is predicated on treatment theory 
or hypotheses will support more targeted measurement and/or the interpretation of evaluation 
findings within a theoretical construct (Orwin, 1998). The logic modeling process assists the 
knowledge-generating activity in identifying the underlying treatment services theory by defining 
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the assumptions and the linkages among the assumptions, treatment services activities, and the 
treatment services outcomes. 

Providing the process evaluation framework. The logic model provides the 
framework for the process evaluation by delineating all of the treatment service elements that 
must be documented so as to fully understand the treatment. For knowledge-generating 
activities, a fully developed logic model describes the treatment services in detail and identifies 
the data that should and should not be collected as part of the process evaluation. 

Providing the framework for integrating process and outcomes analyses. Similarly, 
the logic model provides the framework for integrating the evaluation analysis components. As 
demonstrated, the process evaluation measures the knowledge-generating treatment service 
activities while the outcome evaluation measures the treatment service short-term and long-term 
outcomes associated with the knowledge-generating activities. The analyses, however, are not 
conducted separately or in isolation of each other. Rather, the process evaluation analyses 
provides the context for interpreting the outcome results as well as a conduit for identifying the 
need to change the treatment design and/or the treatment activities. The outcome analysis 
provides “red flags” when short-term or long-term outcomes are different than expected. In 
addition, the process evaluation will provide critical variables and data that will need to be added 
to the outcome database for analysis. 

Ensuring that the knowledge-generating goals and objectives are met. The CSAT 
KD&A program establishes the goals and objectives for the knowledge-generating treatment 
services. The logic model is useful to CSAT, as well as the other stakeholders, in ensuring that 
the overall CSAT goals and objectives are accomplished since the logic model is constructed on 
the basis of a logical sequencing of treatment services so as to achieve the goals and objectives. 

3. KNOWLEDGE-GENERATING SUBSTANCE ABUSE TREATMENT 

OUTCOMES 

In addition to providing the framework for the process evaluation, the logic model guides 
the outcomes evaluation. First, the logic model clarifies the treatment services goals that are 
relevant to the treatment services outcomes, identifies measurable outcomes, and delineates the 
target population and the treatment interventions. 
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Second, if the control or comparison group to be used in the evaluation outcomes analysis 
is receiving alternative treatment services, the logic model is a critical tool to identify the 
similarities and differences between the treatment services being evaluated and the alternative 
treatment services. These differences in treatment service provisions form the basis for the 
expected differences between the treatment and comparison group differences (Conrad et al., 
1998) and will shape the process and outcome evaluation designs. 
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V. Using Logic Models to Specify Knowledge-generating 

Evaluation Measures 



The logic modeling process can be used to develop a framework for specifying the 
evaluation questions, measures, data sources/instruments, and data collection time points. 

Within the Integrated Evaluation Methods approach, this framework is called a data map, and its 
purpose is to identify and clarify the evaluation data requirements. 

1. DATA MAP STRUCTURE AND UTILITY 

A data map very clearly lays out the specifications for evaluation data and the data 
collection plans. The data map provides a bridge between the logic model and the data collection 
activities and establishes the infrastructure for the data analysis plans. It is used to show 
substance abuse treatment services staff the evaluation data requirements and the rationale for 
these requirements. Data maps provide a structured format to answer the primary, secondary, 
and tertiary questions that treatment providers most often ask the evaluation staff: “Why do we 
have to collect these data?” and “How will you use the data once they are collected?” 

2. DATA MAP DEVELOPMENT 

The five steps in developing a data map for a knowledge-generating activity include: 

(1) identifying the over-arching evaluation questions; (2) developing the secondary and tertiary 
evaluation questions; (3) identifying the measures needed to answer the primary, secondary, and 
tertiary evaluation questions; (4) identifying the data sources and instruments; and (5) 
determining the data collection time points. 

A sample data map that parallels an IEM initiative is presented in Exhibit VI- 1 at the 
conclusion of this paper. The sample data map in Exhibit VI-1 is designed to be used as a 
template for creating the evaluation plan for a knowledge-generating activity using the IEM 
package. Therefore, the following paragraphs describe the steps to creating a data map using the 
sample data map as an example. The sample data map is illustrative of the data mapping 
process; it is not intended to be comprehensive. In reality, the data map must be tailored to the 
specific knowledge-generating activity with the evaluation objectives and questions reflective of 
the specific activity. 
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2.1 Identifying Over-Arching Evaluation Questions 

The over-arching evaluation questions are statements, in question form, that must 
ultimately be answered by the evaluation. These questions should be tied directly to the 
knowledge-generating activity goals and objectives and to the logical relationship (logic model 
structure) between the conditions, activities, and outcomes. Exhibit VI- 1 demonstrates how the 
logic model structure guides the development of the over-arching questions: 



■ Conditions for establishing the knowledge-generating activity are identified and 
assessed by the basic, over-arching question: What is the design for the knowledge- 
generating substance abuse treatment service? 

■ Activities necessary for generating knowledge about the substance abuse treatment 
service are identified and assessed by the questions: 

- How was the knowledge-generating substance abuse treatment service 
implemented? 

- How does the treatment service relate to the original design? 

- Who did the knowledge- generating substance abuse treatment service serve? 

- What were the resource requirements and costs of the knowledge-generating 
substance abuse treatment service? 

■ Outcomes, both short-term and long-term, are assessed by answering the question: 

What were the knowledge-generating substance abuse treatment service 
outcomes? and long-term outcomes are further assessed by answering the question: 
What is the relationship between the costs and outcomes of the knowledge- 
generating substance abuse treatment service? 

As shown in the example, the over-arching questions are presented at the top of each sub-section 
in the matrix. Again, these basic, over-arching questions, while germane to all knowledge- 
generating treatment services, should be substantively adapted to the specific knowledge- 
generating goals, objectives, and study questions. 

2.2 Defining Knowledge-Generating Secondary Evaluation Questions 



Secondary evaluation questions are precise, measurable statements (in question format) 
of what the evaluation intends to achieve and are based on the over-arching evaluation questions 
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and on the treatment service objectives. Secondary evaluation questions provide a way to 
incrementally measure achievement of the knowledge-generating goals and objectives and 
provide a linkage to the overarching evaluation measures. Secondary evaluation questions 
address the substance abuse treatment services effort, or process, and cover treatment services 
operations, service delivery, and use of resources. Secondary evaluation questions also address 
treatment effectiveness, or outcomes, and assess the treatment services’ impact on clients and/or 
service delivery network. Evaluation objectives and questions should be stated in terms that can 
be measured. For example, to answer the question “What is the appropriate length of treatment 
in terms of outcomes for different individuals and family units?” measures for treatment delivery 
and treatment outcomes are needed. (Tertiary evaluations reflect a further refinement and 
specification of the evaluation process. These same “rules” apply to the development of tertiary 
evaluation questions.) 

2.3 Identifying Evaluation Measures 

Once the secondary (and tertiary) evaluation questions are defined, the measures needed 
to answer the questions can be specified. Collectively, the evaluation measures provide 
information on treatment service design, implementation, operations, service delivery, costs, 
client behavior, client attitudes, and client experiences. These measures require information 
about the service delivery unit; the clinician background and therapeutic approach; treatment 
costs; and client characteristics, behaviors, and attitudes. Examples of measures needed to 
answer the primary, secondary, and tertiary evaluation questions are specified in Exhibit VI- 1. 
This process supports the ultimate determination of whether the treatment services goals are met. 
Each of the measures identified in the sample data map are further defined and operationalized 
with data definitions and response categories from the IEM companion document: Minimum 
Evaluation Data Set: Core Data Lists. (See the appendix to this document.) 

2.4 Identifying Data Sources/Instruments 

Substance abuse treatment services data are typically obtained from the service delivery 
unit (including the provider director, clinician, and financial management staff) and from the 
clients. Systems level data about the environment in which the treatment services operate are 
obtained from linkage partners, collaborating agencies, and other community agencies or 
community-based data sources. For each of the data sources, appropriate data collection 
instruments are necessary and may include survey formats, interview formats, and protocols to 
abstract records-based data and databases (criminal activity, employment, etc.). Examples of 



?SAT\CTRT_END\lEM\CONCEPT\LOGlCMDL\DATA\LOGICMDL.WPD 



NEDTAC, Page 18 



Using Logic Models to Specify Knowledge-Generating Evaluation Measures 



data sources and types of instruments are presented in Exhibit VI- 1 . The data 
sources/instruments listed in the sample data map (Exhibit VI- 1) are more fully described and 
discussed in the IEM companion document: Guide to Process Evaluation for Substance Abuse 
Treatment Services. (See the appendix to this document.) 

2.5 Determining Data Collection Time Points 

Generally, systems and SDU data are collected at baseline, quarterly, and annually; and 
client-level data are collected at specified intervals. The IEM advocates the collection of client- 
level data at four time points, including: intake (treatment entry), within treatment, treatment 
exit, and at follow-up (3, 6, 12, and/or 18 months following treatment exit). It is recognized, 
however, that data collection activities are the most resource intensive of all evaluation 
components and the ability of the evaluation to conform to the IEM recommended approach is 
directly linked to the evaluation budget and resources. 

An important role for the evaluation planning process is the capability to adjust the 
evaluation components, as needed, when faced with new information. Thus, after the data map 
has been developed, the knowledge-generating evaluation questions will probably need to be re- 
examined to clarify and finalize the evaluation plan. 
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The list below provides a summary of recommended activities to aid in the development 
of logic models: 



■ Clarify the knowledge-developing substance abuse treatment services from the 
perspective of the KD or KA program, treatment services managers and staff, 
evaluators, and other key stakeholders 

■ Explore treatment services “reality,” including the plausibility and measurability of 
treatment-site services goals and objectives in light of the knowledge-generating 
treatment goals and objectives 

■ Involve intended users of evaluation information to determine the knowledge- 
generating evaluation priorities and intended uses of evaluation information on 
substance abuse treatment services performance 

■ Apply logic model techniques to knowledge-generating substance abuse treatment 
service planning and implementation and to evaluation planning and implementation 

■ Apply logic model techniques to data map development to ensure that the 
specification of knowledge-generating evaluation questions, measures, and variables 
are linked to the treatment goals and objectives and the knowledge-generating goals 
and objectives. 

The development of logic models is critical to ensuring that the knowledge-generating substance 
abuse treatment services evaluation can be carried out in a way that will yield accurate and useful 
information to document treatment effectiveness and improve treatment services and activities. 

A review of the activities listed above shows the close relationship between the development of 
the logic models, planning the treatment services, and planning the treatment services evaluation. 
Applying the ideas presented in this paper will improve the process of substance abuse treatment 
services evaluation that support knowledge generation and lead to the acquisition of new 
knowledge so as to identify exemplary treatment services and to realize systemic and treatment 
services improvements. 
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Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Exhibit VI- 1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Exhibit VI-1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Exhibit VI- 1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Exhibit VI-1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Exhibit VI-1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Exhibit VI-1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Exhibit VI-1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 





CSAT\CTRT_END\IEM\CONCEPT\LOGICMDL\DATA\LOGICMDL.WPD 



O 

40 



G'D 



NEDTAC, Page 30 



BEST COPY AVAILABLE 



Exhibit VI- 1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 





csat\ctrt_end\iem\concept\logicmdl\data\logicmdl.wpd 



cv 

lh 






NEDTAC, Page 3 1 



BEST COPY AVAILABLE 



Exhibit VI-1 (Continued) 

Sample Data Map for Knowledge-Generating Substance Abuse Treatment Evaluation 
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Appendix: 

Integrated Evaluation Methods Package: 

A Guide For Substance Abuse Treatment 
Knowledge-Generating Activities — Executive Summary 

Since its inception, the Center for Substance Abuse Treatment (CSAT) has provided 
Federal leadership to improve substance abuse treatment accessibility, effectiveness, and 
efficiency. CSAT’s mission and activities have evolved from directly supporting treatment 
services to supporting knowledge-generating activities. This evolution is evident in the current 
Substance Abuse and Mental Health Services Administration policy on evaluation as described 
in Evaluation Policy, SAMHSA, 1995. 

The need for an integrated model of evaluation and planning at SAMHSA is presented in 
“Evaluation in the Substance Abuse and Mental Health Services Administration,” Evaluation 
and the Health Professions, by Marsh, Jansen, Lewis, & Straw, 1 996. CSAT also supports site- 
specific, cross-site, and national evaluations that have provided experience with a wide array of 
evaluation design and implementation methods. These experiences further supported the need 
for an integrated evaluation strategy and led to the development of a comprehensive set of 
evaluation products, including concept papers, technical assistance (TA) materials, and analytic 
tools. Collectively, these products are referred to as the Integrated Evaluation Methods (IEM) 
Package. The IEM Package organizes these products within an evaluation framework that is 
designed to support CSAT knowledge development and application goals. The evaluation 
framework itself was constructed on the basis of accumulated experiences among internationally 
known treatment service evaluation professionals. The IEM Package reflects and incorporates 
evaluation experiences gained over the past decade. 

Evaluation Framework and the Integrated Evaluation Methods Package 

National evaluation experiences have reinforced the fact that substance abuse treatment 
evaluation involves a standard set of tasks that generally occur in the following order: 

■ Planning the evaluation/knowledge-generating activities, which includes selecting 
the substance abuse treatment issue, identifying the theoretical foundation for the 
intervention, determining knowledge development program goals and implementation 
approach, and setting the evaluation goals and objectives that determine the overall 
parameters of the evaluation 
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■ Selecting the evaluation design, which sets forth the overall strategy for establishing 
the process and outcome evaluation questions, measurement approach, and 
generalizability of findings 

■ Developing the data requirements, which flow from the evaluation questions and 
measures and include: SDU, clinician, cost, and client data 

■ Developing data collection instruments, which are based on the data requirements 
and are developed or selected from an integrated inventory of instrumentation 

■ Collecting the data, which includes developing data management processes and tools 
(including quality control procedures) and conducting the data collection activities 

■ Analyzing the data, which involves multiple levels of comparison and is governed 
by an analysis plan 

■ Reporting the evaluation findings, which includes evaluation knowledge 
dissemination and application within the field. 

The evaluation process outlined above provided a framework for the development of products 
related to these evaluation concepts and methods. Taken together, those products comprise the 
IEM Package. 

Integrated Evaluation Methods Products 

CSAT requested the development of a series of evaluation concept papers, TA materials, 
and tools to support and operationalize each phase in the evaluation of substance abuse treatment 
knowledge-generating activities. These items are included in the IEM Package. The concept 
papers are based on theoretical evaluation research constructs that have been adapted to 
substance abuse treatment services evaluation and knowledge-generating activities. The concept 
papers primarily support the evaluation planning phase and address such topics as the self- 
adjusting treatment evaluation model, cost analyses, and performance measurement. The TA 
materials and tools include specific evaluation methods that have direct applicability to substance 
abuse treatment knowledge-generating activities. The concept papers and TA materials that 
constitute the IEM Package are listed and briefly described in Exhibit I. 
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Evaluation Framework and Integrated 
Evaluation Methods Package 



Executive Summary ? 



EVALUATION 

FRAMEWORK 


INTEGRATED EVALUATION METHODS PRODUCTS 


1. Planning the 
evaluation/ 
knowledge- 
generating 
activities 


■ Integrated Evaluation Methods: A Guide for Substance Abuse Treatment 
Knowledge Generating Activities: Concept paper that describes the development of an 
evaluation framework, evaluation concepts, and TA materials to support the framework. 

■ Self-Adjusting Treatment Evaluation Model: Concept paper that describes an 
approach for integrating evaluation findings within treatment operations so as to adjust 
and improve service delivery. 

■ Building Team Capability to Fully Implement and Utilize the Self-Adjusting 
Treatment Evaluation Model: Concept paper to assist treatment providers in building 
capabilities to integrate the self-adjusting treatment model within day-to-day operations 
and service delivery. 

■ Adding “Value” to CSAT Demonstrations: The What, How and Why of Cost 
Analysis: Concept paper on the need for and types of cost analyses for CSAT 
demonstrations and knowledge-generating activities. (The Lewin Group) 

■ Performance Measurement for Substance Abuse Treatment Services: Concept 
paper about the increasing importance of provider performance measurement and 
analyses and an explanation of the case-mix adjustment methodology. 

■ Client Levels of Functioning as a Component of Substance Abuse Treatment 
Services Evaluation: Description of the rationale and methods for assessing client level 
of functioning and recommended core LOF data elements that could help to measure the 
effectiveness of treatment services received. 

■ Substance Abuse Treatment Evaluation Policy Notebook: These materials are aimed 
at facilitating understanding of the SAMHSA policy for evaluation and federal 
regulations on client confidentiality and assisting evaluators to meet CSAT evaluation 
requirements. 

■ Substance Abuse Treatment Evaluation Resource Notebook: The notebook contains 
evaluation bibliographies and listings of organizations, hot lines, on-line data bases, and 
contact information for obtaining assistance in evaluating treatment services. 


2. Selecting the 
evaluation design 


■ A Guide to Process Evaluation for Substance Abuse Treatment Services: TA tool 
presenting purposes of process evaluation and the application of process evaluation 
methods to single site and multi-site treatment services. 

■ Using Logic Models in Substance Abuse Treatment Evaluations: TA tool describing 
logic model purposes and techniques for designing and planning the evaluation of 
treatment services. 

■ A Guide to Selecting an Outcome Evaluation Design for Substance Abuse 
Treatment Evaluations: TA tool describing overall strategies for developing 
evaluation questions, measurement, controls, validity/reliability, sampling, design 
effects, and generalizability of findings. (Battelle) 
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Exhibit I (Continued) 
Evaluation Framework and Integrated 
Evaluation Methods Package 



Executive Summary 



evaluation 

framework 


INTEGRATED EVALUATION METHODS PACKAGE 


3. Developing data 
requirements 


■ Minimum Evaluation Data Set (MEDS): Core Data Lists: TA tool for developing a 
uniform set of variables and response categories for the service delivery unit (SDU), 
clinician, cost, and client evaluation measures. 

■ Substance Abuse Treatment Cost Allocation and Analysis Template (SATCAAT): 
User manual to analyze treatment costs by unit of service for an SDU. (Capital 
Consulting Corporation) 


4. Developing data 
collection 
instruments 


* 

■ Substance Abuse Treatment Services Evaluation Data Collection Instruments: Data 
collection instruments that fully incorporate the MEDS and that have been field tested 
for validity and reliability, as follows: Service Delivery Unit (SDU) Description; 
Clinician Background and Practice Survey; protocols to collect Adult, Adolescent and 
Child (in treatment with parent) Client Data at Intake, During Treatment, at Treatment 
Discharge and Post Treatment; Adult and Adolescent Record Extraction forms; and a 
section on protection of human subjects and informed consent. 


5. Collecting the 
data 


■ Staying In Touch: A Fieldwork Manual of Tracking Procedures for Locating 
Substance Abusers for Follow-up Studies (UCLA): User manual to establish and 
implement client follow-up data collection systems and procedures. 

■ Strategies for Follow-up Tracking of Juvenile, Homeless, and Criminal Justice 
System-Involved Substance Abusers: Overview and Bibliographies, 1990-1998: 
Description of tracking techniques used to increase response rates for follow-up 
interviews with homeless and juvenile/criminal justice involved substance abusers. 


6. Analyzing the 
data 


■ A Guide to Substance Abuse Treatment Evaluation Data Analysis: Recommended 
methods and procedures for analyzing process, SDU, clinician, cost, and client 
evaluation data. 


7. Reporting the 
evaluation 
findings 


■ Substance Abuse Treatment Evaluation Product Outlines Notebook: Compendium 
of outlines for evaluation products including evaluation plans, interim evaluation reports, 
final evaluation reports, replication studies, case studies, and ethnographies. 
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CSAT Evaluation “Stakeholders” 

Evaluation “stakeholders” are individuals, groups, or organizations that have a significant 
interest in how well a program or activity functions. (See P.H. Rossi, H.E. Freeman, & M.W. 
Lipsey, Evaluation: A Systematic Approach, 6th Edition, 1999.) Within the context of the IEM 
Package, CSAT evaluation stakeholders include CSAT senior managers, CSAT project officers, 
and CSAT grantees and contractors including treatment service providers, coordinating centers, 
study sites, site-specific evaluators, and national evaluators. 

Utility of the IEM Package for CSAT Evaluation Stakeholders 

While the conceptual and TA materials were developed from the perspective of the site- 
specific and multi-site evaluator, the concepts and TA tools have important utility for CSAT 
managers, project officers, and treatment service providers. The stakeholder’s position 
determines the perspective and utility of the IEM Package concepts and tools. For example, a 
CSAT senior manager can use the IEM Package to acquire a comprehensive evaluation context 
for planning and funding the knowledge-generating activities, the project officer can use the IEM 
Package to ensure that GFA/RFP applications are complete and include a full complement of 
design, execution, and product components, and the site-specific and multi-site evaluators can 
use the IEM Package to ensure that evaluation designs, data collection plans, data analyses, and 
product development have a consistent evaluation framework and compatible data across 
program areas. The suggested utility of the IEM Package for CSAT evaluation stakeholders is 
summarized in Exhibit II. n, 
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Executive Summary 



Exhibit II 

Utility of IEM Package for CSAT Evaluation Stakeholders 



STAKEHOLDERS 


ROLES AND RESPONSIBILITIES 


IEM PACKAGE UTILITY 


SENIOR 

MANAGERS 


■ Policy development 

■ Issue identification for KD&As 

■ Grant/contract funding decisions 

■ Overall program management 

■ Sustainability 

■ Dissemination 

■ Long-term strategic planning 

■ Program designs 

■ KA activities 


■ Comprehensive evaluation framework 

■ Comprehensive evaluation components 

■ Roles and responsibilities for local/national 
evaluators as well as CSAT/grantee staffs 

■ Guidance for evaluation designs and 
products 

■ Standardized evaluation measures 

■ Logic models for program and evaluation 
design 


PROJECT 

OFFICERS 


■ GFA/SOW development 

■ Grant/contract application review 

■ Grant/contract monitoring 

■ Knowledge-generating products 

■ Identification and replication of 
promising practices 

■ Technical assistance assessment 


■ Guidelines for high-quality evaluation 
designs (process and outcome) 

■ Logic models for program and evaluation 
designs 

■ List of evaluation measures with 
instrumentation 

■ Guidelines for evaluation products 


GRANTEES: 
STUDY SITES 


■ Grant applications 

■ Project development, implementation 

■ Local evaluation management 

■ Local evaluation coordination 

■ Knowledge-generating product 
development 


■ Evaluation plan outline 

■ Process and outcomes evaluation designs 

■ SDU, clinician, cost, and client measures 

■ Roles and responsibilities for grantee 
provider/evaluator staff 

■ Guidelines for evaluation products 


GRANTEES: 

MULTI-SITE 

EVALUATORS 


■ Grant applications 

■ Comprehensive evaluation designs 

■ Evaluation implementation: 

- Data collection 

- Data analysis 

- Reporting evaluation findings 

■ Evaluation product development 


■ Evaluation concepts 

■ Logic models 

■ Evaluation designs 

■ Evaluation data requirements 

■ Data collection instrumentation 

■ Data collection process and procedures 

■ Data analysis 

■ Product development 


NATIONAL 

EVALUATORS/ 

SERVICES 

RESEARCHERS 


■ Contract applications 

■ Comprehensive evaluation designs 

■ Evaluation implementation: 

- Data collection 

- Data analysis 

- Reporting evaluation findings 

■ Evaluation product development 


■ Evaluation concepts 

■ Logic models 

■ Evaluation designs 

■ Evaluation data requirements 

■ Data collection instrumentation 

■ Data collection process and procedures 

■ Data analysis 

■ Product development 



1 EM products and other evaluation materials may be obtained from: 
http://neds.calib.com 
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